Jul 19
2
Google posted this morning that they are going to stop unofficially supporting the noindex, nofollow and crawl-delay directives within robots.txt files. Google has been saying this for years actually and hinted this was coming really soon and now it is here
Originally posted here:
Google To Drop Any Support For crawl-delay, nofollow, and noindex in robots.txt